WHAT IS CLAIMED IS: 



1. A system for synchronizing multi-modal 
interactions, comprising : 

a ' multi-modal application comprising at least a first 
mode process and a second mode process; 

a multi-modal shell for managing and synchronizing 
information exchanges between the first and second mode 
processes of the multi-modal application; and 

an API (application program interface) that allows the 
first and second mode processes to register their respective 
active commands and corresponding actions with the multi- 
modal shell. 

2. The system of claim 1, further comprising a 
registry having a registration table, managed by the multi- 
modal shell, that comprises a list of each of the registered 
commands and corresponding synchronized actions that results 
in both the first and second mode processes upon execution 
of a registered command by one of the first and second mode 
processes . 
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3. The system of claim 1, further comprising at least 
a first mono-mode application and a second mono-mode 
application, wherein the multi-modal shell manages and 
synchronizes information exchanges between the first and 
second mono-mode applications. 

4. The system of claim 1, further comprising at least 
a first device having a first user interface modality and a 
second device having a second user interface modality, 
wherein the multi-modal shell manages and synchronizes 
information exchanges between the first and second devices. 

5. The system of claim 4, wherein the first device, 
second device and multi-modal shell are distributed over a 
network, and wherein the API is implemented using 
distributed APIs or protocols. 

6. The system of claim 2, wherein the API comprises a 
mechanism for converting a mono-mode application to a multi- 
modal application . 
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7. The system of claim 6, wherein the mono-mode 
application is a GUI application, and wherein the mechanism 
provides speech enablement of the GUI application by 
registering the active commands of the GUI application and 
building a grammar for the registered commands to support 
the commands in a speech modality. 

8. The system of claim 2, wherein the API comprises a 
mechanism for building a multi-modal application. 

9. The system of claim 8, wherein the mechanism is 
used for directly programming the registry by building a 
registration table having user-defined commands and 
corresponding actions for each of the modalities of the 
multi-modal application . 

10. The system of claim 1, further comprising an 
operating system, wherein the multi-modal shell executes on 
top of the operating system. 

11. The system of claim 1, wherein the system is 
distributed over a network. 
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12. The system of claim 1, wherein the multi-modal 
application is a multi-mocial browser, wherein the first mode 
process renders a first modality and the second mode process 
renders a second modality. 

13. The system of claim 12, wherein the first modality 
is GUI and the second modality is speech. 

14. The system of claim 12, wherein the multi-modal 
shell processes a CML (conversational markup language) file 
to send modality- specific presentation information in the 
•CML file to the respective browsers. 

15. The system of claim 14, wherein the CML file 
encapsulates the modality-specific presentation information 
in a single modality-independent representation. 

16. The system of claim 14, wherein the CML file 
comprises a combination of declarative markup languages. 

17. The system of claim 16, wherein the CML file 
comprises a single file combining the declarative markup 
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languages and synchronization elements to provide tight 
synchronization between the declarative markup languages. 



18. The system of claim 16, wherein the CML file 
5 comprises a separate file for each of the declarative markup 

languages and wherein the separate files are loosely 
synchronized at predefined points. 



19. A method for synchronizing multi-modal 
10 interactions, comprising the steps of: 

receiving a command in a first modality; 
triggering (i) an action in the first modality and (ii) 
a corresponding action in at least a second modality, based 
on the received command; and 
15 updating application states or device states associated 

with the first modality and the second modality. 



20. The method of claim 19, further comprising the 
steps of: 

20 registering active commands associated with the first 

modality and active commands associated with the second 
modality; 
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associating, with each registered command of the first 
modality, an action on the first modality and a 
corresponding action on the second modality; and 

associating, with each registered command of the second 
5 modality, an action on the second modality and a 

corresponding action on the first modality. 

21. The method of claim 20, further comprising the 
step of building a command-to-action registration table 

10 based on the registered commands and actions. 

22. The method of claim 21, wherein the registration 
table is built by a multi-modal shell via API calls from the 
applications or devices associated with the first and second 

15 modalities. 



23. The method of claim 20, wherein the step of 
triggering comprises the steps of: 

looking up the received command in the registration 
20 table; and 

executing the actions associated with the received 
command on the first and second modalities. 
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24. The method of claim 20, further comprising the 
steps of: 

registering a callback handle for each of the 
registered commands to notify the first and second 
5 modalities of completion of the actions corresponding to the 

registered commands. 

25. The method of claim 24, wherein the step of 
updating the application states or the device states 

10 comprises the steps of executing the callback handle 

associated with the received command to trigger a callback 
action on the first modality and a callback action on the 
second modality. 

15 26. The method of claim 19, wherein the step of 

triggering comprises the steps of: 

executing first thread associated with the received 
command; and 

triggering a corresponding second thread to initiate 
20 the corresponding action on the at least second modality. 



27. The method of claim 26, wherein the threads are 
applets . 
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28. The method of claim 26, wherein the threads 
communicate via socket connections. 

29. A program storage device readable by a machine, 

5 tangibly embodying a program of instructions executable by 

the machine to perform method steps for synchronizing multi- 
modal interactions, the method comprising the steps of: 
receiving a command in a first modality; 
triggering (i) an action in the first modality and (ii) 
10 a corresponding action in at least a second modality, based 

on the received command; and 

updating application states or device states associated 
with the first modality and the second modality. 

15 30. The program storage device of claim 29, further 

comprising instructions for performing the steps of: 

registering active commands associated with the first 
modality and active commands associated with the second 
modality; 

20 associating, with each registered command of the first 

modality, an action on the first modality and a 
corresponding action on the second modality; and 
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associating, with each registered command of the second 
modality, an action on the second modality and a 
corresponding action on the first modality. 

31. The program storage device of claim 30, further 
comprising instructions for performing the step of building 
a command-to-action registration table based on the 
registered commands and actions. 

32. The program storage device of claim 30, wherein 
the instructions for performing the step of triggering 
comprise instructions for performing the steps of: 

looking up the received command in the registration 
table; and 

executing the actions associated with the received 
command on the first and second modalities. 

33. The program storage device of claim 30, further 
comprising instructions for performing the steps of: 

registering a callback handle for each of the 
registered commands to notify the first and second 
modalities of completion of the actions corresponding to the 
registered commands. 
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34. The program storage device of claim 33, wherein 
the instructions for performing the step of updating the 
application states or the device states comprise 
instructions for performing the step of executing the 
callback handle associated with the received command to 
trigger a callback action on the first modality and a 
callback action on the second modality. 

35. The program storage device of claim 29, wherein 
the instructions for performing the step of triggering 
comprise instructions for performing the steps of: 

executing first thread associated with the received 
command; and 

triggering a corresponding second thread to initiate 
the corresponding action on the at least second modality. 

36. The program storage device of claim 35, wherein 
the threads are applets. 

37. The program storage device of claim 35, wherein 
the threads communicate via socket connections. 
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